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Abstract. We introduce a new approach for cryptanalysis of key agree- 
ment protocols based on noncommutative groups. This approach uses 
functions that estimate the distance of a group element to a given sub- 
group. We test it against the Shpilrain-Ushakov protocol, which is based 
on Thompson's group F, and show that it can break about half the keys 
within a few seconds on a single PC. 
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1 Introduction 

Key agreement protocols have been the subject of extensive studies in 
the past 30 years. Their main task is to allow two parties (in the sequel, 
Alice and Bob) to agree on a common secret key over an insecure com- 
munication channel. The best known example of such a protocol is the 
Diffie-Hellman protocol, which uses a (commutative) cyclic group. Over 
the last few years, there was a lot of interest in key agreement protocols 
based on noncommutative groups, and much research was dedicated to 
analyzing these proposals and suggesting alternative ones (see, e.g., [1, 
4-8, 10-12], and references therein). 

A possible approach for attacking such systems is the length-based 
cryptanalysis, which was outlined in [6] . This approach relies on the exis- 
tence of a good length function on the underlying group, i.e., a function 
i{g) that tends to grow as the number of generators multiplied to obtain 
g grows. Examples of groups known to have such length functions are the 
braid group B]\[ [2] and Thompson's group F [3]. For these groups, several 
practical realizations of length-based attacks were demonstrated [4, 5, ?]. 
These attacks can achieve good success rates, but usually only when we 
allow the algorithm to explore many suboptimal partial solutions, which 
greatly increases both the time and space complexities (see [5] for more 
details). 



Wc introduce a novel approach to cryptanalysis of such key agreement 
protocols, which relies on the notion of subgroup distance functions, i.e., 
functions that estimate, for an element g G G and a subgroup H < G, 
the distance from g to H. The motivation for these distance-based at- 
tacks is the fact that several families of public key agreement protocols 
suggest predefined pairs of subgroups of the main group to be used for 
key generation, and their security depends on the ability of the adver- 
sary to generate any elements in these subgroups, which are in some 
way equivalent to the originals (see [?,11]). We construct the theoretical 
framework for distance-based attacks and demonstrate its applicability 
using the Shpilrain-Ushakov protocol in Thompson's group F [12] as an 
example. Although it has recently been shown by Matucci [8] that the 
implementation of the proposed protocol in F can be broken determin- 
istically using a specialized attack based on the structural properties of 
the group, it is still an interesting test case for more generic attacks, such 
as the one proposed here. 

The paper is organized as follows: in Section 2 we present the protocol 
in its general form. We then introduce in Section 3 the notion of subgroup 
distance function and a general attack scheme based on it. Section 4 
describes the setting for the protocol in Thompson's group F. In Section 5 
we introduce several subgroup distance functions in F. Section 6 describes 
our experimental cryptanalytic results. 

2 The Shpilrain-Ushakov Key agreement Protocol 

The protocol below was suggested by Shpilrain and Ushakov in [12]. The 
authors suggested to use Thompson's group F for its implementation. 
Before we focus on that example, we'll discuss the general case. 

(0) Alice and Bob agree (publicly) on a group G and subgroups A,B<G, 
such that ah = ha for each a & A and each b £ B. 

1. A public word z G G is selected. 

2. Alice selects privately at random elements ai G A and bi E B, com- 
putes ui = aizbi, and sends ui to Bob. 

3. Bob selects privately at random elements a2 E A and 62 G 5, com- 
putes U2 = b2za2, and sends U2 to Ahce. 

4. Alice computes Ka = aiM2&i = 0162^0261, whereas Bob computes 
Kb = b2Uia2 = b2aizbia2. 

As 0162 = b2ai and 0261 = 6ia2, = Kb = K and so the parties 
share the same group element, from which a secret key can be derived. 



2.1 Breaking the protocol 

The goal of the adversary is to obtain the secret group clement K from 
the publicly known elements ui, U2 and z. For this it suffices to solve the 
following problem: 

Definition 1 (Decomposition problem) Given z ^ G and u = azb 

where a G A and b G B, find some elements d € A and b € B, such that 
dzb = azb. 

Indeed, assume that the attacker, given ui = aizbi, finds di G A and 
61 G B, such that dizbi = aizbi. Then, because 112 = 622:02 is known, the 
attacker can compute 

diU2bi = 01622:0261 = 62012:6102 = b2Uia2 = Kb ■ 

Alternatively, the attacker can break the protocol by finding a valid de- 
composition of U2 = 622:02. 

For any given d E A we can compute its complement b = z~^d~^u = 
z''^d~^{azb), which guarantees that dzb = azb. The pair a,6 is a solution 
to this problem if, and only if, 6 € S. A similar comment applies if wc 
start with b € B. This involves being able to solve the group membership 
problem, i.e., to determine whether b E B (or d E A in the second case). 

It should be stressed that solving the decomposition problem is suffi- 
cient, but not necessary in order to cryptanalyze the system. All that is 
required in practice is finding some pair a, 6 that succeeds in decrypting 
the information passed between Alice and Bob. Any pair d E A and b E B 
will work, but there can be other pairs, which are just as good. This ob- 
servation can be useful in cases where the group membership problem is 
difficult or in groups where the centralizers of individual elements are con- 
siderably larger than the centralizers of the subgroups (which is not the 
case in F, see [9]). For simplicity, in the sequel we will restrict ourselves 
to solutions where d E A and b E B. 

3 Subgroup distance functions 

Definition 2 (Subgroup distance function) Let G be a group, H < 
G a subgroup. A function du '■ G — > M"*" is a subgroup distance function 

if it satisfies the following two axioms: 

1. Validity: dnih) = for all h E H . 

2. Non-triviality: dnig) > for all g ^ H. 



It is an invariant subgroup distance function if it also satisfies: 
(3) Invariance: dnigh) = dnihg) = dnig) for all g E G and h E H. 

Clearly, if it is possible to evaluate a subgroup distance function djj on 
all elements of G, then the membership decision problem for H is solvable: 
g € H <^=^ dnig) = 0. Conversely, if one can solve the membership 
decision problem, a trivial distance function can be derived from it, e.g., 
duig) = 1 — XHig), where xh is the characteristic function of H. 

Obviously, this trivial distance function is not a good example. For 
the subgroup distance function to be useful, it has to somehow measure 
how close a given element g is to H, that is, if dnidi) < c?//(52), then 
gi is closer to H than g2- This concept of "closeness" can be hard to 
define, and even harder to evaluate. The notion of what's considered a 
good distance function may vary, depending on the subgroups and on the 
presentation. In the sequel we will discuss concrete examples of subgroup 
distance function in Thompson's group F. 

Assuming the existence of such functions, consider the following algo- 
rithm for solving the decomposition problem: 

Algorithm 1 (Subgroup distance attack) 

We are given words z, xzy G G, where x E X and y E Y , X,Y are com- 
muting subgroups of G and Sx, Sy are their respective (finite) generating 
sets. The goal it to find some x G X and y G Y , such that xzy = xzy. 
The algorithm runs at most a predefined number of iterations N. 

1. Let a; <— 1. 

2. For each gi G compute Xi = xgi, its complement yi = z~^x^^xzy 
and evaluate d.Y{yi). If dyiyi) = 0, let x = Xi, y = yi and halt. 

3. Let j be the index of the minimum dyiyi) (if several such j are pos- 
sible, choose one arbitrarily). 

4. If the maximal number of iterations N has been reached, terminate. 
Otherwise, let x <— xj and return to step 2. 

Observe that if the algorithm halts in step 2, then the pair x,y is a 
solution of the decomposition problem. 

Algorithm 1 is very similar to the length-based attacks described in [4, 
?]. The difference is that it uses the subgroup distance function, instead 
of the length function to evaluate the quality of candidates. As such, any 
extensions applicable to the length-based algorithms (such as memory, 
lookahead, etc.) can be used with the distance-based attack as well. Refer 
to [5, ?] for more information. 



3.1 Attacking the Shpilrain-Ushakov protocol 

The adversary is given the common word z and the public elements ui,U2- 
These can be translated into four equations in the group: 

Ul = aizbi 

U2 = b2Za2 

u^^ = b'^^z^^a^ 
= a2^z~^b2 

Algorithm 1 (with or without possible extensions) can be applied to 
each of the four equations separately, thus attacking each of the four 
private elements ai, 02, 6^^, 63 ^. A single success out of the four attempts 
is sufficient to break the cryptosystem (see Section 2.1). 

4 Thompson's group 

Thompson's group F is the infinite noncommutative group defined by the 
following generators and relations: 

F = {xo,xi,X2,... I x'^XkXi = Xk+i {k>i)) (2) 

Remark 1 Prom Equation (2) it's evident that the elements xo,xi and 
their inverses generate the entire group, because x^^ = XQ~'^xf^XQ~^ for 
every k >2. 

Definition 3 A basic generator xf^ of F is called a letter. A generator 
Xi is a positive letter. An inverse x~^ is a negative letter. A word in F 

is a sequence of letters. We define \w\ as the length of the word w , i.e., 
the number of letters in it. 

Definition 4 A word w E F is said to be in normal form, if 

w = Xi^--- Xi^xJ^^ ■ ■ ■ x'J^ (3) 
and the following two conditions hold: 
(NFl ) ii < ■■■ <ir and ji < ■ ■ ■ < jt 

(NF2) If both Xi,x~^ occur in w, then at least one of Xi+i,x~^^ occurs too. 
A word is said to be in seminormal form if only ( NFl ) holds. 



(1) 



While a seminormal form is not necessarily unique, a normal form is, 
i.e., two words represent the same group element if and only if they have 
the same normal form [3]. The following rewriting rules can be used to 
convert any word to its seminormal form [12]: 

For all non-negative integers i < k: 



(Rl) XkXi XiXk+i 
{R2) Xj^ Xi XiXf^_^_i 
{R3) x~^Xk Xfc+iXj"^ 

For all non-negative integers i: 

(i?5) x-^Xi ^ 1 

The seminormal form can be subsequently converted to a normal form 
by searching for pairs of indices violating (NF2), starting from the bound- 
ary between the positive and negative parts, and applying the inverses of 
rewriting rules (Rl) and (R4) to eliminate these pairs [12]: 

Suppose that {xi^,xj^^) is a pair of letters violating (NF2) and that 
a and b are maximal with this property (i.e., there exists no violating 
pair {xii^,xj^^) with k > a and I > b). Then ia = jb and all indices in 
^ia+i ' ' ' ^ir^Jf^ ■ ■ ■ -^j^+i higher than ia + 1 (by definition of (NF2)). 
Applying the inverse of (Rl) to Xi^ and the inverse of (R4) to xj^^ we get: 

ft 1 1 ry . , , , nf . ( nf . ... If • If . . . T* nf . . . If • 

W — Xjj •I'ta K-^la+l -^l-r-^jt jb+l' jb 

c 

V nff . , , , 'y ■ 1 • • • If • -t ( T* ■ nf '\ nf • • • T* • • • Hf • 

cancel ^ ^ 
— > ■ ■ ■ Xj^.i (a^j^+i-l • ■ ■ Xi^-iXj^_i ■ ■ ■ Xj^^^_^) Xji^_^ ■ ■ ■ Xj^ 



The violating pair (xj^ , xj^^ ) is cancelled and the subword d obtained 
from c by index shifting contains no violating pairs (by the assumption 
of maximality on {a,b)). Thus, we can continue searching for bad pairs, 
starting from a — 1 and 6—1 down. Thus we are guaranteed to find and 
remove all the violating pairs and reach the normal form. 

Definition 5 (Normal form length) For w F, whose normal form 
is w, define the normal form length as Inf{w) = \w\. 



The following lemma shows the effect multiplication by a single letter 
has on the normal form of the word. This result will be useful in the 
following sections. 

Lemma 1 Let w G F and x = xf^ he a basic generator of F in the 
presentation (2). Then iNwixw) = £nf{w) ± 1 (and due to symmetry, 
iNpi'wx) = £nf{w) ± 1). 

Proof. We 'II concentrate on the product xw ( obviously, the case of wx 
is similar) and observe what happens to the normal form of w when it's 
multiplied on the left by the letter x. Without loss of generality, w = 
Xjj ■ • ■ Xif,x~^ ■ ■ ■ xj^ is in normal form. Denote the positive and negative 
parts of w by Wp and Wn respectively. 

Assume that x = xt is a positive letter. Then hw is converted to a 
seminormal form by moving x into its proper location, while updating its 
index, using repeated applications of (Rl). Assuming m applications of 
(Rl) are necessary, the result is of the form: 

hw = Xi^ ■ ■ ■ Xi^Xt^rn^im+i ' ' ' •^''■k-^ji ' ' ' "^ji ' 

where im <t + m — 1 and im+i >t + m. 

Remark 2 Observe that it is not possible that im = t + m — 1, because in 
order to apply (Rl): xt+m-iXi^ — > Xi^xt+m, one must have im < t+m—1. 

Example 1 w = x^xrxuXg^x^^ , b = xg. hw = xg ■ x^xrxuXg^x^^ is 
converted to bw = x^x^xioxuXg^ x'^^ , by 2 applications of (Rl). 

Obviously, hw is a seminormal form and \hw\ = \w\ + 1. If hw is in 
normal form (as in the above example), we're done. The only situation 
where it's not in normal form, is if it contains pairs violating (NF2). 
Since xt+m is the only letter introduced, the only violating pair can be 

miXi_^m) ■ This may occur, if w contained x^j^^, but neither Xf-\.rn; 
nor xfl^^^. 

Example 2 w = x^xjXuXg^x^^ , b = xj. hw = xj ■ x^xrxuXg^x^^ 
is converted to hw = xsXjXgXnXg^x^^ . In this case {xg,Xg^) violates 
(NF2). The inverse of (Rl) is applied to rewrite xqXu — > xiqXq, and 
xgXg^ are canceled out, yielding the (normal) word bw = x^x^xiox^^ . 

Whenever a situation occurs as described above, the pair (a;t+„i , ,t^„J 
is cancelled, according to the procedure described in Section 4- This causes 
all indices above t + m to he decreased by 1. The resulting word is 

hw = Xi^--- Xi^Xi^^^-i ■ ■ ■ Xi^-iXj^^^ ■ ■ ■ Xj^^^_iXj^ ■ ■ ■ Xj^ , 



where im < t+m—1, im+i ^ t+m+2, jn < t+m and jn+i > t+m+2. We 
have \bw\ = \w\ — 1 and, in fact, bw is in normal form. Indeed, once the 
pair {xt-i-m,x^^^) is cancelled, the only new pair violating (NF2) that 
can he introduced is (xt+^-i, hut this is not possible, because 
does not appear in bw, due to Remark 2. This completes the proof 
for positive letters. 

Now, consider the case where negative letter, bw is con- 

verted to a seminormal form by moving x^^ to the right, while updating 
its index, using the different rewriting rules. There are two possible out- 
comes: 

(1) After m applications of (R2) the resulting word is 

bw = Xi^ ■ • • Xi^x^j^.^Xi^_^^ ■ ■ ■ Xif^Xj^ ■ ■ ■ Xj^ , 

where im+i =t + m, and so the pair is cancelled by applying (R5). Now, 
because im < t-\-m — l, the elimination of the pair (xt+m, ^t+m) ^^^^ '^^^ 
introduce pairs that violate (NF2), and so bw is in normal form and has 
\bw\ = \w\ — 1. 

Example 3 w = xsxyxg^xj^, b = x^^. bw = Xq^x^xjXq^x'^^ is con- 
verted to and the pair of inverses is cancelled out to obtain 

3^4 ^ 3^32^9 X^ . 

(2) x^^ is moved to its proper place among the negative letters, up- 
dating its index if necessary. This is completed through m applications of 
(R2), followed byk — m applications of (R3) and finally, l — n applications 
of (R4), to obtain 

bw = Xi^ ■ ■ ■ Xi^x^_^^Xi^_^_j^-\.i ■ ■ ■ Xi^,+iXj^_^_-^ ■ ■ ■ X ■^^^_^-^x^_^^x ■ ■ ■ x-^ , 

where im < t + m — 1, im+i > t-\-m, jn+i > t+m and jn < t+m. Because 
the letter xt+m is not present in bw ( otherwise the previously described 
situation would occur), the newly introduced letter x^_^^ cannot violate 
(NF2), and therefore bw is in fact in normal form and \bw\ = \w\ + 1. 

Example 4 u) = x^x^Xg ^x^^ , b = x'^^. bw = x'^^x^xix^^x^^ is rewrit- 

i/GTl CIS' «jCg X'jXq X^ ^ X^X^Xq iZ^g ^ "^3*^8*^10 

This completes the proof for negative letters. 

□ 



4.1 The Shpilrain-Ushakov protocol in Thompson's group 

For a natural number s > 2 let Sa = {xo^i^, . . . , xo-x^^}, Sb = {xg+i, ■ ■ ■ , 
X2s} and Sw = {a^Oj • • • Sw generates F (see Remark 1). Denote 

by As and Bg the subgroups of F generated by Sa and Sb, respectively. 

All of the following facts are shown in [12]: Ag is exactly the set of 
elements whose normal form is 

rff . , , , /y . If . . . -If* 

i.e, has positive and negative parts of the same length m, and additionally 
satisfies ik — k < s and jk — k < s for every A; = 1, . . . , m. 5^ is the set of 
all elements of F whose normal form consists only of letters with indices 
> s + 1. Additionally, Ag and Bg commute elementwise, which makes 
them usable for implementing the protocol in Section 2. 

Key generation Let s > 2 and L be positive integers. The words ai, 02 G 
Ag, 61,62 G Bg, and w € F arc all chosen of normal form length L, as 
follows: Let X be A, B, or W. Start with the empty word, and multiply 
it on the right by a generator (or inverse) selected uniformly at random 
from the set Sx- Continue this procedure until the normal form of the 
word has length L. 

For practical and (hopefully) secure implementation of the protocol, 
it is suggested in [12] to use s G {3, 4, . . . , 8} and L G {256, 258, . . . , 320}. 

5 Subgroup distance functions in Thompson's group 

In this section we'll suggest several natural distance functions from the 
subgroups Ag,Bg < F defined in Section 4.1. These distance functions 
can be used to implement the attack outlined by Algorithm 1. 

5.1 Distance functions from Bg 

For w E F define Pi{w) and Ni{w) as the number of occurrences of Xi 
and x^^ in the normal form w of w. 

Definition 6 (Distance from Bg) Let s <2 be an integer. For w E F 
the distance from Bg is defined as 

s 

dB.(«^) = 5^(P«(«^) + A^i(«^)) 

i=0 



Claim 1 is a distance function. 



Proof. This is immediate, since an element is in Bg if and only if its 
normal form does not contain generators with indices below s + 1 (see 
Section 4-1)- ^ 

Claim 2 ds^ is an invariant distance function. 

Proof. It is enough to consider only the generators of Bg ■ Indeed, if mul- 
tiplication by a single generator of Bg does not change the distance of a 
word w, neither does multiplication by a sequence of these generators. 

Let w E. F. Let b = xf^^, where a > 0. By Lemma 1, we know 
that b is either moved to its proper position ( and £]\fF{bw) = (.mf{w) + 1) 
or it is cancelled with its inverse, either by (R5) or as part of a pair 
violating (NF2), in which case £]\[F{bw) = (,nf{w) — 1. The index of b 
is initially above s, and may only increase when the rewriting rules are 
applied. Therefore, if b is cancelled at some point, the index of its inverse 
is also above s. Furthermore, when pairs of elements are rewritten, the 
lower-indexed element is not affected, so any letters with indices < s will 
not be affected by moving b. Finally, if b is cancelled out due to violating 
(NF2), the process again only affects letters with indices higher than b's 
(see the proof of Lemma 1). In all cases, the generators with indices < s 
are not affected at all, and so dB^ibw) = ds^iw). 

□ 

One can intuitively feel that ds^ is a natural distance function, because 
it counts the number of "bad" letters in w (letters that do not belong to 
the subgroup Bg). Indeed, if w is in normal form, w = WpWcWn, where Wp 
and Wn are the "bad" positive and negative subwords, respectively, then 
dssi'w) = \wp\ + \wn\ and Wp^ww~^ € B. 

We now introduce another natural function that measures distance 
from Bg. 

Definition 7 (Weighted distance from Bg) Let s < 2 be an integer. 
For w E F the weighted distance from Bg is defined as 

s 

d^^{w) =Y,{s + l-i) {Pi{w) + Ni{w)) 

1=0 

dss does not only count the "bad" letters, but assigns a score for each 
letter, depending on how far below s + 1 it is (in particular, dB^iw) < 
dBsiw) for all w € F. The following claim is straightforward. 



Claim 3 ds^ is an invariant distance function. 

Proof. The proof of Claim 2 shows that multiplication by b does not alter 
any letters below s + 1 in w. Therefore, the weight of each such letter is 
also preserved. □ 



5.2 Distance functions from As 

We will now describe a number of natural distance functions from the 
subgroup Ag. Recall (Section 4.1) that Ag is the set of all elements in F, 
whose normal form is of the type Xi^ ■ ■ ■ Xi^xJ^ ■ ■ ■ xj^, i.e, has positive 
and negative parts of the same length m, and additionally satisfies ik — k< 
s and jk — k < s for every k = 1, . . . ,m. 

Definition 8 (Distance from Ag) Let s > 2 be an integer. Let w & F, 
such that its normal form is w = Xi^ ■ ■ ■ Xi^xJ^ ■ ■ ■ xj^ . The distance from 
As is defined as 

dAsi'w) = \{k : ik - k > s}\ + \{l : ji - I > s}\ + \p - n\ 

dAsiut) is the number of "bad" letters in w, i.e., letters that violate 
the Ag property, plus the difference between the lengths of the positive 
or negative parts. dA, is clearly a distance function. However, it is not 
invariant, as shown by the following example: 

Similarly we can define a weighted distance function from Ag, which 
not only counts the number of bad letters, but gives a score to each such 
letter, based on the difference ik — k (or jk — k). 

Definition 9 (Weighted distance from Ag) Let s > 2 be an integer. 
Let w G F, such that its normal form is w = Xi^ ■ ■ ■ Xi^xJ^ ■ ■ ■ xjK The 
weighted distance from Ag is defined as 

ik-k>s jk-k>s 

dA,M= ^ (ik-k-s + l) + (jk-k-s+l) + \p-n\ 

k=l...p k=l...n 

For each bad letter Xi^ or xj^, dA^ adds a positive integer. As such, 
it's a distance function, which is again not invariant (the example above 
works here too). 

A somewhat different approach to defining distance from Ag arises 

from the observation that the number of bad letters can be less important 
than the maximum value of the differences ik — k and jk — k across the 
word, which measures the size of the violation. The difference between 



the two distance funetions roughly corresponds to the difference between 
the Li and L^o norms. 

Let w = Xi, ■ ■ ■ Xi x~^ ■ ■ •x7^- Suppose that for some integer k we 
have ik — k — s + 1 = nip > and that nip is the maximum for all ifc. 
By multiplying the word by x™*" we shift the position for all the original 
positive letters of w by nip, and so all of the positive letters, including the 
first m xo's have ik — k < s. Similarly, if m„ is the maximum violation in 
the negative subword, multiplication by Xq^" on the right eliminates all 
violations among negative letters. However, this still does not mean that 
the word is in As, because the positive and negative lengths may differ. Let 
w' be the normal form obtained from w through multiplication by x^^ and 
Xq on the left and right, respectively. Let Ip and Z„ be the corresponding 
lengths of the positive and negative parts of w'. If Ip — In > 0, then 
w'x^Q G As- lilp — ln < 0, then Xg" ''''w' G Ag. Altogether, any word can 
be changed to a word in Ag through multiplication by nip + nin + \lp + ln\ 
indices (when Ip and Z„ are evaluated after multiplying by x^^ and x^"*" ) . 

This observation suggests the following distance function: 

Definition 10 (Maximum-based distance from Ag) Let s > 2 be 

an integer. Let w E F, such that its normal form is w = Xj^ • ■ ■ Xi^ 
x~^ ■ ■ ■ x~^. Let 

Jn Jl 

rup = max ({0} U {i^ — k — s + l:k = l... p}) 

and 

rUn = max ({0} U {jfc — k — s + 1: k=l... n}) . 
The maximum-based distance from Ag is defined as 

(^^{w) = nip + rUn + Hp + nip) - (n + m„)| 

For every w E Ag nip, nin and \p — n\ are by definition, while for 
every w ^ Ag at least one of them has to be positive, so the cU^^ is a 
distance function. It turns out that, unlike the two previously defined 
distance functions, is also invariant. 

Claim 4 is an invariant distance function. 

Proof. As with Claim 2, it's sufficient to prove that multiplication by a 

single generator of Ag does not change the distance from any word w to 
Ag. We will consider multiplications on the left by generators and their 
inverses. The multiplication on the right follows symmetrically. 



Let w = Xi-^ ■ ■ ■ Xi^x-^ ■ ■ ■ Xj^ , without loss of generality, in normal 
form. Consider the generator xqx^^, where 1 < t < s. Define w' as the 
normal form of xqx^^w. For the parameters p,n,mp,mn ofw, denote by 
p' , n' , m'p, m'^ their corresponding values in w' . 

Prom Lemma 1 it follows that each of the letters x^^ and xq can either 
be cancelled out with the appropriate inverse, decreasing the length by 1, 
or placed in its appropriate location, increasing the length by 1. There is 
a total of 4 possible options: 

(1) x'^^ is cancelled out, but xq is not: w' = xqXi^ ■ ■ ■ Xi^Xi^j^^ ■ ■ ■ 
XipXj^ ■ ■ ■ , where x~[^^ is cancelled out with Xi^^^ after m applications 
of (R2). Lt follows that p' = p, n' = n and m'^ = nin (because the negative 
letters are unaffected). Observe also that there can be no bad letters among 
the first m: indeed, (R2) is applied m times, for each k = 1 . . . m rewriting 
•^t+fc-i'^*fe ~^ •^ik-'^t+k' necessarily i^ < t+k — 1 for all k, or equivalently, 
ik — k<t — l<s. The multiplication by xq on the left only increases 
their relative positions, thus decreasing i^ — k. Now, any possible bad 
letters above i^ are unchanged, and neither is their relative position, so 
m'p = nip and overall d^^{w') = d^^{w). 

(2) Both x^^ andxQ are cancelled, out: w' = Xj^-i • • • Xi^^iXi^^^-i ■ ■ ■ 
Xij^^ixJ^_^ ■ ■ ■ x~^^^_^Xq~'^ . Here p' = p — 1, n' = n — 1 and m'^ = nin 

because all negative letters xj^ with jk > had both their indices and 
their relative positions decreased by 1. The same thing applies to positive 
letters above im, which are the only positive letters that may be bad. So 
again, m'p = rup and d^^{w') = d^^{w). 

(3) Neither x^^ , norxQ are cancelled out: w' = xoXi^ ■ ■ ■ Xj^Xj^^^^+i • • • 
Xip+ixJ^^^ ■ ■ ■ xJ^l^^^Xt_^^xJ^^ ■ ■ ■ xj^^. Here p' = p + I andn' = n + l. 
Due to the former observation, bad positive letters may only exist beyond 
the first m. All these letters had their indices ik and their relative po- 
sitions k increased by 1, so the difference is preserved and m'p = mp. 
Among the negative letters, only the letters whose indices increased, also 
had their relative position increased, so jk — k is preserved for all the orig- 
inal letters of w. Hence, m'^ > mn and the only situation when it may 
actually increase is when the new maximum is attained at the new letter, 
i.e., m'^ = (i + m) — (gr + 1) — s + 1 > m„. Because t < s, m < p and 
q < n, we have m'^ < p — q, from which it follows that 

(p'+mp) — (n'+m^) = {p' -n')-\-{m'p—m'jj) = — (n+l)+mp— m(j > 

> mp + ip — n) — {p — q) = mp + q — n > 



Assuming m'^ > nin, it's obvious that 

(p-n) + {nip - m„) > {p' - n') + (m^ - m'^) > , 

and so ifirin increases, |(p + nip) — (n + m„)| decreases by the same amount, 
and overall d^^{w') = d'^^{w). 

(4) is not cancelled out, but xq is: w' = • • • Xi^^iXi^^^^ ■ ■ ■ 

^ip^Jn ■ • • ^7,+i^t+rr^-i2^7,-i ■ • • ^7/+i-i2^o"^ w'^ere p' =p,n' = n, m'^ = 
rup (because the first ni positive letters, whose indices have changed, con- 
tained no bad letters), and m'^ again may only increase, if it's attained 
at x^_^j^_i. Repeating the same calculations shows that {w') = d^ {w) 
in this case too. 

Now consider the inverse xtXQ^ and denote w' = xtXQ^w. The four 
possible outcomes are: 

(1) Xq^ is cancelled out, but xt is not: x^^ can only be cancelled out if 
i\ = 0, and the resulting word is: w' = Xi^ ■ ■ ■ Xi^Xt+m—iXi^_^i ■ ■ ■ ^i^xj^ ■ ■ ■ 

. Here p' = p, n' = n, m'^ = nin (negative part is not affected) and 
m'p = nfip because the letters Xi^ to Xi^ cannot be bad and the relative 
position of other positive letters has not changed. 

(2) Both Xq^ and xt are cancelled out: Assuming xt is cancelled out 
(due to violation of (NF2)) with , w' = Xi^- ■ ■ xi^Xi^j^-^^-i- ■ ■ Xi^-i 

x~l-i ■ ■ ■ -^7+1-13^7,^1 • • • xj^^- Here p' = p - I, n' = n - I, ni'^ = nip, 
because Xi^ to Xi^ cannot be bad and the relative position of other positive 
letters has not changed, and m'^ = nin, because the letters whose positions 
shifted also had their indices decreased. 

(3) Neither Xq^, nor Xt are cancelled out. w' = Xii+2 • • • Xi^+2Xt+m 
3^i,„+i+i • • • Xi^+ixJ^^^ ■ ■ ■ x~\-^x~''. Herep' = p+1, n' = n+1, m'^ = nip, 
because indices above im grew by 1, as did their positions, and indices 
ii,...,im cannot be bad, and also m'^ = mn, because all letters whose 
indices increased (jq and above) shifted in position accordingly. 

(4 ) Xq^ is not cancelled out, but xt is: w' = a:jj_|_2 • • • Xi^+2Xi^^;^ ' ' ' Xi^ 
x~^ ■ ■ ■ x~^ x~\_ 1 1 • • • x~\-,Xq'^, the cancelled pair being {xt+m,x~^), where 
jq = t + m. In this case, any positive letters that can be bad kept their 
indices and positions, the negative letters jV+i, • • • ,jg-i had their indices 
and positions shifted, while the letters jq+i, ■ ■ ■ ,jn kept their indices and 
positions. So m'p = nip and m'^ = m„ and obviously p' = p and n' = n. 

We see that in all the possible cases, d^^{w') = d^^{w). This completes 
the proof. □ 



6 Experimental results 



To test the applicability of the subgroup distance functions to cryptanal- 
ysis, we tested Algorithm 1 against the Shpilrain-Ushakov protocol in the 
settings of Thompson's group. Initially, each of the five distance functions 
presented in the previous section was tested separately: we generated a 
public element azb and tried to recover a single private element a or 6 
from it. For the recovery of a, the functions and ds^ were used to 
assess the quality of the complements. Similarly, for the recovery of b, we 
tried , dA^ and . 

For each distance function, the experiment was run at least 1000 times, 
each time with new, randomly generated keys, with the minimum recom- 
mended parameters of s = 3, L = 256. The bound N = 2L was chosen 
on the number of iterations, since preliminary experiments have shown 
that the success rates do not increase beyond that. The results are sum- 
marized in Table 1. It can be seen that the distance functions ds^ and 
noticeably outperform the other distance functions, in recovering a 
and b, respectively. The fact that clearly outperforms its counter- 
parts suggests that the notion of invariance may be useful for assessing 
the suitability of a given distance function. 

Table 1. Success rates for the different subgroup distance functions 













dZ 


Recovery probability 


11.7% 


3.4% 


3.7% 


3.4% 


23.3% 



Preliminary experiments have shown that, regardless of the settings, 
the success probability of finding ai given aizbi is similar to that of finding 

given a^^z~^6^^. A similar assertion holds for 62 and b^^ . Therefore, 
in order to estimate the overall success rate against an actual instance of 
the cryptosystem, it's sufficient to try to recover one of the four a's and 
6's. If we denote by pa and pb the probability of successfully recovering 
a and b, respectively, and assume that all probabilities are independent, 
then, the expected total success rate is roughly 1 — (1 — Pa)^(l — Pb)^ 
(because each instance of the protocol contains two elements of type a 
and two of type b). 

When the success rates of the two best distance functions, for a 
and for b, are combined, the expected overall success probability, ac- 



cording to the above, is between 50% and 54%, which was experimentally 
verified. Note that this attack is very efficient, since it involves no back- 
tracking, no lookahead, and no analysis of suboptimal partial results: it 
tries to peel off the generators by a greedy algorithm, which considers only 
locally optimal steps. Attacking each key required only a few seconds on 
a single PC, and it is very surprising that such a simple attack succeeds 
about half the time. These results are much better than those achieved 
by length-based attacks of similar complexity on this cryptosystem (see 
[9]). 

It is interesting to note that possible extensions of the attack, such as 
memorizing many suboptimal partial solutions or using significant looka- 
head (which require much higher time and space complexities) have dif- 
ferent effects on length-based and distance-based attacks. While it was 
shown in [9] that these extensions greatly improve the success rates of 
the length-based attack, experiments with the distance-based attack, with 
similar values of the memory and lookahead parameters, showed almost 
no improvement. However, the situation may be very different for other 
cryptosystems and other subgroup distance functions. 

To further test the performance of the distance functions, several ex- 
periments were run with different values of the parameters {s,L). We 
used the combination of ds, and d^^, which was established as the best 
in the former experiment. Table 2 shows the overall success probabil- 
ity, for L G {128,256,320,512,640,960} and s € {3,5,8}. The success 
rates stay remarkably consistent across different lengths for a given s, 
and even increasing s does not cause a significant drop. The time com- 
plexity of the attack grows linearly with s and roughly quadratically with 
L, with most of the time being spent on computing normal forms of ele- 
ments in the group. For the largest parameters presented here, the attack 
still required under a minute in most cases. This suggests that for the 
Shpilrain-Ushakov cryptosystem the distance-based attack remains a vi- 
able threat, even when the security parameters s and L are increased 
beyond the original recommendations. 



Table 2. Success rates for different combinations of (s, L) 





L = 128 


L = 256 


L = 320 


L = 512 


L = 640 


L = 960 


s = 3 


51.7% 


47.9% 


55.5% 


51.2% 


50.4% 


52.6% 


s = 5 


46.0% 


47.1% 


48.4% 


51.1% 


48.2% 


48.3% 


s = 8 


36.2% 


42.8% 


41.3% 


46.5% 


42.4% 


50.3% 



7 Conclusion 



We introduced a novel form of heuristic attacks on public key cryptosys- 

tems that are based on combinatorial group theory, using functions that 
estimate the distance of group elements to a given subgroup. Our results 
demonstrate that these distance-based attacks can achieve significantly 
better success rates than previously suggested length-based attacks of 
similar complexity, and thus they are a potential threat to any cryp- 
tosystem based on equations in a noncommutative group, which takes 
its elements from specific subgroups. It will be interesting to test this 
approach for other groups and other protocols. 
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